Papers with crosslingual transfer
TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data (2025.coling-main)
Copied to clipboard
| Challenge: | Existing mPLMs that handle non-transliterated data are not sufficient to train crosslingual models. |
| Approach: | They propose a framework to transliterate related languages into a common script by exploiting existing mPLMs and their tokenizer without any training. |
| Outcome: | The proposed framework can create strong baselines for data that is transliterated into a common script by exploiting an existing mPLM and its tokenizer without any training. |
LangSAMP: Language-Script Aware Multilingual Pretraining (2025.acl-long)
Copied to clipboard
| Challenge: | Recent multilingual pretrained language models often avoid using language embeddings, which places a significant burden on token representations to encode all language-specific information. |
| Approach: | They propose a method that incorporates both language and script embeddings into the output of Transformer blocks before passing the final representations to the language modeling head for prediction. |
| Outcome: | The proposed method outperforms the baseline model in zero-shot crosslingual transfer across diverse downstream tasks. |
TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | The world’s more than 7000 languages are written in at least 293 scripts, which poses a difficulty for multilingual pretrained language models in learning crosslingual knowledge through lexical overlap. |
| Approach: | They propose a framework that optimizes the Transliteration Contrastive Modeling objective to fine-tune an mPLM by contrasting sentences in its training data and transliterations in a unified script. |
| Outcome: | The proposed model outperforms Glot500-m on zero-shot crosslingual transfer tasks while retaining uniformity across scripts. |
Analyzing Bayesian Crosslingual Transfer in Topic Models (N19-1)
Copied to clipboard
| Challenge: | a theoretical analysis of crosslingual transfer in probabilistic topic models is presented . we use Gibbs sampling to quantify the loss of knowledge across languages . |
| Approach: | They propose a method to quantify the loss of knowledge across languages during crosslingual transfer in probabilistic topic models. |
| Outcome: | The proposed model quantifies the loss of knowledge across languages during this process . it is validated on a diverse set of five languages and discusses best practices for data collection and model design . |
On Difficulties of Cross-Lingual Transfer with Order Differences: A Case Study on Dependency Parsing (N19-1)
Copied to clipboard
| Challenge: | Existing studies on crosslingual transfer have focused on word-level information sharing, but words are not independent in sentences; their combinations form larger linguistic units, known as context. |
| Approach: | They propose to use orderagnostic models to transfer word order to distant languages . they train dependency parsers on an English corpus and evaluate their transfer performance on 30 other languages. |
| Outcome: | The proposed model performs better on languages with different word orders than on other languages. |
Argument Mining in Data Scarce Settings: Cross-lingual Transfer and Few-shot Techniques (2024.acl-long)
Copied to clipboard
| Challenge: | Recent work on sequence labelling has explored different strategies to mitigate the lack of manually annotated data for the large majority of the world languages. |
| Approach: | They propose to use the mask objective to exploit the few-shot capabilities of pre-trained language models to improve their performance. |
| Outcome: | The proposed model-transfer outperforms data-transference and fine-tuning outperformed few-shot methods for Argument Mining task. |
Discrete and Soft Prompting for Multilingual Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | In few-shot learning, discrete and soft prompting perform better than finetuning in multilingual cases. |
| Approach: | They show that discrete and soft prompting perform better than finetuning in crosslingual transfer and in-language training of multilingual natural language inference. |
| Outcome: | The proposed prompting model outperforms finetuning in crosslingual transfer and in-language training of multilingual natural language inference. |